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1.  Introduction 


1.1  Background 

This  report  is  the  second  in  a  series  addressing  Human-Systems  Integration  (HSI) 
support  provided  by  the  US  Army  Research  Laboratory’s  (ARL’s)  Human 
Research  and  Engineering  Directorate  (HRED)  to  the  Brigade  Modernization 
Command  (BMC)  for  the  Network  Integration  Evaluations  (NIEs).  (Note;  HSI 
was  previously  referred  to  as  MANPRINT  [Manpower  and  Personnel 
Integration].  MANPRINT  was  the  Army’s  formal  program  for  HSI.)  The  NIEs 
are  a  series  of  semiannual  exercises  (identified  by  fiscal  year)  intended  to 
integrate  and  mature  the  Army’s  tactical  networks  in  an  operational  context. 
During  an  NIE,  the  Army  also:  1)  conducts  integrated  and  parallel  operational 
tests  of  selected  Army  programs  of  record,  2)  evaluates  developmental  and 
emerging  network  capabilities  in  an  operational  environment,  and  3)  assesses 
non-networked  capabilities  in  an  integrated  operational  environment.  The  first 
report  on  this  subject  (Hawley  2014)  addressed  HSI  support  provided  to  the  BMC 
for  NIEs  conducted  during  Fiscal  Year  (FY)  2013,  with  an  emphasis  on  NIE  13.2. 
The  primary  focus  of  the  following  discussion  is  HSI  results  from  the  FY  2014 
series  of  NIEs. 

ARE  HRED  participated  in  the  FY  2014  series  of  NIEs  in  3  capacities.  First, 
HRED  personnel  provided  HSI  support  to  the  Army  Test  and  Evaluation 
Command  (ATEC)  during  formal  operational  tests  of  individual  equipment  items. 
Second,  HRED  personnel  supported  the  Director,  System  of  Systems  Integration 
and  Engineering  (SoSI&E)  in  the  evaluation  of  selected  system-of-systems  used 
within  the  exercise.  In  present  usage,  a  system-of-systems  is  a  collection  of  task- 
oriented  systems  that  are  integrated  to  create  a  new,  more  complex  system  that 
offers  more  functionality  and  performance  than  the  simple  sum  of  the  component 
systems.  And  third,  HRED  personnel  from  the  Fort  Bliss,  Texas  Field  Element 
provided  support  to  the  BMC  for  the  evaluation  of  individual  equipment  items 
and  system-of-systems  used  within  a  broader  team  and  unit  context.  The  emphasis 
of  much  of  the  discussion  to  follow  is  this  third  level  of  HSI  support. 

A  Command  Post  (CP)  is  an  example  of  a  system-of-systems  as  that  term  is  used 
above.  The  CP  consists  of  a  number  of  individual  component  systems,  each 
designed  to  address  a  single  aspect  of  the  mission  command  warfighting  function. 
The  combination  of  these  components  within  a  single  entity  (the  CP)  forms  a 
more  complex  system-of-systems  that  if  structured  properly  offers  more 
functionality  and  capability  than  the  simple  sum  of  the  individual  components. 
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Historically  within  the  Army,  HSI  has  been  applied  at  the  individual  system  level 
for  programs  of  record  (e.g.,  the  individual  eomponent  systems  eomprising  a  CP). 
Applieation  of  HSI  at  the  broader  system-of-systems,  team,  and  organizational 
levels  is  a  relatively  new  undertaking.  A  large-seale  exereise  sueh  as  the  NIEs 
permits  such  macrolevel  HSI  work  to  be  performed.  The  unit  of  HSI  interest  at 
the  system-of-systems  level  is  an  integrated,  humans-plus-machines  suite  used  to 
support  a  speeifie  warfighting  funetion — for  example,  mission  eommand.  At  the 
organizational  level,  the  foeus  of  HSI  analyses  is  the  impaet  of  individual 
eomponents  and  system-of-systems  used  within  the  broader  unit  and  mission 
eontext.  In  NIE  terminology,  these  equipment  sets  are  referred  to  as  “Capability 
Sets”.  A  primary  interest  at  this  third  level  of  HSI  assessment  is  the  aggregate 
organizational,  personnel,  and  training  impact  associated  with  the  introduetion  of 
new  Capability  Sets  (CSs). 

1.2  Approach 

HRED’s  first  look  at  the  system-of-systems  level  of  HSI  support  referenced  above 
was  during  NIE  13.1.  After  observing  field  operations  and  reviewing  NIE 
database  entries  ereated  during  that  exereise,  HRED  staff  members  eoneluded  that 
the  eognitive  load  assoeiated  with  mission  eommand  was  emerging  as  an  HSI 
eoneern.  (Note:  Entries  in  NIE  databases  are  provided  by  military  experts  on 
individual  systems,  system-of-systems,  or  eoneepts,  and  are  not  direetly  related  to 
cognitive  load.  However,  they  often  indireetly  refer  to  cognitive-load-related 
issues.)  Eollow-on  eonversations  with  personnel  from  other  organizations 
supporting  the  NIEs  eonfirmed  this  observation.  Consequently,  the  primary  foeus 
of  HRED’s  HSI  support  to  the  BMC  during  NIE  13.2  was  cognitive-load  issues 
assoeiated  with  mission  eommand.  In  present  usage,  eognitive  load  is  defined  as 
the  aggregate  mental  load  plaeed  on  eommanders,  key  battle  staff  members,  or 
other  personnel  by  an  inereasingly  eomplex  mission  eommand  work  setting.  As  a 
eonstruct  impaeting  mission  eommand,  eognitive  load  is  developed  in  additional 
detail  in  Hawley  (2014). 

Prior  to  the  start  of  NIE  14.1,  the  BMC  Commanding  General  (CG)  tasked  ARE 
HRED’s  HSI  support  team  to  eonduet  what  he  termed  a  CP  “Ease  of  Use” 
assessment.  The  objeetive  of  the  CP  Ease  of  Use  assessment  was  to,  “Evaluate 
solutions  that  simplify  and  proteet  the  network  while  enhaneing  the  eommand 
post  by  deereasing  its  physieal  eomplexity  and  eognitive  burden  on  eommanders 
and  staff”  The  BMC  CG  further  identified  4  subordinate  analysis  issues  to  frame 
this  overall  objeetive: 
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1)  Identify  cognitive  systems  engineering  issues  impacting  the  horizontal  and 
vertical  integration  of  mission  command  systems  within  CPs  as  configured 
inNIE  14.1. 

2)  Assess  the  impact  of  Soldier  New  Equipment  Training  (NET)  on  system 
capability  understanding.  Also  determine  the  effects  of  NET 
implementation  within  the  tactical  context  as  an  enabler  for  commander 
decision  making.  Make  recommendations  for  beneficial  changes. 

3)  Assess  Soldier  NET  from  a  “CP-as-a-System/Platform”  perspective  and  its 
implementation  within  the  tactical  context.  Provide  recommendations  for 
beneficial  changes. 

4)  Evaluate  complexity  and  cognitive  load  within  CPs  as  configured  in  NIE 
14.1.  Provide  specific  examples  of  sources  of  complexity  and  cognitive 
load  in  current  CP  operations.  Evaluate  cognitive  “pressure  points.” 
Provide  recommendations  to  reduce  complexity  and  cognitive  burden. 

2.  NIE  14.1  Data  Collection  and  Reporting 


2.1  Data  Collection 

Data  relevant  to  these  analysis  issues  were  obtained  from;  1)  field  observations  in 
CPs  during  NIE  operations,  2)  interviews  with  commanders  and  battle  staff 
members,  3)  interviews  with  exercise  Observer/Controllers  (0/Cs),  and  4)  a 
review  of  NIE  database  entries.  During  unit  observation  and  interview  sessions, 
HRED  personnel  were  accompanied  by  a  military  escort  officer  (typically  a  major 
or  a  lieutenant  colonel)  provided  by  one  of  the  primary  NIE  support 
organizations.  The  escort  officers  were  experienced  military  operations  research 
analysts  and  were  familiar  with  unit  operations  and  NIE  equipment  and 
objectives.  They  assisted  HRED  personnel  in;  1)  gaining  access  to  unit  CPs,  2) 
making  essential  introductions  to  unit  command  and  battle  staff  personnel,  3) 
understanding  what  was  transpiring  as  the  CP’s  operations  were  observed,  and  4) 
focusing  follow-on  interviews  on  key  aspects  of  cognitive  load  in  mission 
command.  HRED  personnel  also  used  the  escort  officers  after  the  fact  to  assist  in 
making  sense  of  and  clarifying  observations,  conclusions,  and  recommendations. 
The  extensive  literature  base  on  human  factors  applied  to  military  system  design 
provided  a  conceptual  backdrop  for  HSI  data  obtained  during  NIE  14.1. 
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2.2  Reporting  and  Assessment 

Following  NIE  14.1,  ARE  HRED’s  support  team  reported  baek  to  the  BMC  CG 
on  the  results  of  the  Ease  of  Use  assessment.  In  present  eontext,  Ease  of  Use 
assessment  refers  to  a  usability-type  analysis  performed  by  observing  CP 
personnel  in  their  operational  work  setting  and  asking  questions  about  that  work 
while  it  is  being  performed.  More  detailed  postexereise  foeus  group  sessions 
provided  additional  usability  results,  along  with  elarifioation  of  speoifie  work- 
related  observations. 

The  team’s  assessment  suggested  that  3  HSI-related  problem  areas  were  at  the 
root  of  many  of  the  observed  mission  eommand  oomplexity-eognitive  load  issues 
in  CPs  as  eonfigured  during  NIE  14. 1.  The  problem  eategories  are  deseribed  in 
the  following  sections. 

2.2.1  CP  Component  Design 

Many  of  the  individual  systems  (often  referred  to  as  “boxes”  or  “widgets”)  used 
to  support  mission  command  were  neither  user  friendly  or  sufficiently  reliable. 
There  were  numerous  reports  in  the  NIE  database  about  individual  equipment 
items  being  error-prone,  unreliable,  and  not  user  friendly.  Moreover,  similar 
remarks  regarding  the  same  systems  have  been  recorded  across  several  NIEs. 
Another  aspect  of  the  design  problem  concerns  the  “stovepiped”  nature  of  the  way 
in  which  the  mission  command  systems  comprising  CPs  are  developed  and 
evaluated.  A  CP  is  a  system  in  and  of  itself  (i.e.,  a  system-of-systems).  However, 
the  components  comprising  the  CP  are  developed  and  evaluated  mostly  in 
isolation,  and  often  by  different  proponents  and  vendors.  Their  relationship  with 
other  CP  components  is  not  always  considered,  nor  is  their  design  based  on  an 
understanding  of  complex  cognitive  work  in  context.  Consequently,  the  pieces  of 
the  CP  “puzzle”  do  not  always  fit  together  smoothly  to  support  mission  command 
as  an  integrated  warfighting  system.  The  underlying  issue  here  is,  if  the  CP  had 
been  considered  as  an  integrated  system,  would  this  component  have  been 
conceived  and  designed  as  it  was?  This  is  a  hypothetical  “what  if’  issue  above 
and  beyond  the  component  integration  problem  discussed  next.  It  should  be  noted 
that  this  stovepiping  problem  is  not  unique  to  mission  command  systems  or  to  the 
Army. 

2.2.2  CP  Integration 

Many  of  the  individual  systems  within  CPs  were  not  well  integrated  to  support 
mission  command  as  cognitive  work.  When  used  within  the  context  of  a 
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discussion  of  mission  command  complexity  and  cognitive  load,  3  aspects  of 
integration  must  be  considered: 

1)  Physical  Integration,  Physical-technieal  integration  refers  to  the  network 
of  sensor  and  communieations  eapabilities  that  link  users  through  various 
interfaces  and  enable  them  to  acquire  and  share  information.  This  aspeet 
of  integration  primarily  refers  to  mission  command  component 
conneetivity  and  interoperability. 

2)  Operational  Integration.  Operational  integration  refers  to  integrating 
technical  functions  with  the  human  eognitive  proeesses  they  are  intended 
to  support  and  making  that  eognitive  work  more  reliable.  Practically, 
operational  integration  involves  the  incorporation  of  mission  command 
materiel  solutions  into  battle-staff  proeesses  and  proeedures  to  inerease 
their  effectiveness  and  efficieney  in  exeeution.  Achieving  effeetive 
operational  integration  requires  attention  to  issues  sueh  as,  1)  the  design  of 
human  interfaces,  2)  communication  systems  and  practices,  3)  battle-staff 
training,  4)  battle-staff  teamwork,  and  5)  CP  organization  and 
management  proeedures.  Effective  operational  integration  supports  users 
in  making  sense  of  information  transmitted  via  teehnical  conneetions, 
intuitively  understanding  the  implications  of  that  information,  and 
responding  appropriately. 

3)  Perceptual-Cognitive  Integration.  Well-designed  CPs  must  support 
deeision-makers  at  all  levels  with  the  insight  and  foresight  required  to 
make  effeetive  decisions,  to  manage  associated  risks,  and  to  consider 
seeond  and  subsequent  order  effects.  This  involves  the  cognitive  ability 
“at  a  glanee”  to  see  and  understand  a  tactical  situation  and  thereby  enable 
independent  deeision  and  correct  action.  These  capabilities,  often  labeled 
pereeptual-cognitive  integration,  are  the  product  of  both  adequate 
teehnical  integration  and  effeetive  operational  integration  of  mission 
eommand  eomponents  into  the  Military  Decision  Making  Process 
(MDMP)  and  CP  practiees  (Mission  Command  White  Paper  2012). 
Pereeptual-cognitive  integration  is  not  a  proeess  per  se.  Rather,  adequate 
physical  integration  eoupled  with  suitable  operational  integration  create 
the  conditions  for  perceptual-cognitive  integration. 

Mission  command  is  technology-supported  cognitive  work.  Core  mission 
command  activities  are  cognitive  in  nature.  That  is,  the  equipment  is  there  to 
support  human  cognitive  activities  such  as  sensemaking,  mental  projeetion  to  the 
future,  and  deeision  making.  Results  from  NIE  14.1  (and  previous  NIEs) 
suggested  that  physical  and  operational  integration  to  support  mission  command 
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as  cognitive  teamwork  generally  had  not  been  addressed  adequately.  Various 
items  of  equipment  do  not  interoperate  smoothly,  and  operational  integration  of 
new  equipment  items  into  mission  command  procedures  has  not  been  suitably 
addressed.  The  result  is  an  inadequate  level  of  perceptual-cognitive  integration. 
Commanders  find  it  difficult  to  use  mission  eommand  equipment  suites  to  form  a 
composite  “picture”  of  the  battlespace.  Previous  researeh  indieates  that  attempting 
to  integrate  and  use  disparate,  often  incompatible  eomponents  in  a  CP  has  been  a 
problem  at  least  since  the  introduction  of  digital  technology  (Howse  and  Cross 
1999). 

2.2.3  Training,  Practice,  and  Experience 

Many  of  the  personnel  using  mission  command  systems  had  not  been  adequately 
trained  individually  or  as  a  set  (system-of-systems).  Moreover,  battle-staff 
personnel  considered  as  a  team  had  not  been  provided  sufficient  time  to  become 
familiar  with  the  equipment  suites  used  to  support  mission  command  as  a 
warfighting  function.  Trust  in  and  reliance  on  technology  emerge  from  familiarity 
and  positive  experienees  with  that  teehnology.  The  level  of  expertise  required  to 
effectively  use  mission  command  support  technology  cannot  be  developed  as  part 
of  traditional  NET  or  a  short,  follow-on  orientation  program  within  a  receiving 
unit.  Interview  results  indicated  that  hands-on  experience  gained  during  previous 
NIEs  with  equipment  items  and  equipment  suites  mattered  signifieantly  during 
NIE  14.1.  Also,  there  were  numerous  remarks  in  the  NIE  14.1  database 
coneerning  “lack  of  trust”  in  the  equipment  provided  to  NIE  partieipants.  There 
were  suggestions  that  some  of  this  lack  of  trust  derived  from  lack  of  equipment 
familiarity.  NIE  participants  simply  had  not  had  time  to  become  comfortable  with 
mission  eommand  equipment  or  equipment  suites  and  conversant  with  their 
potential  uses. 

The  HSI  team’s  report  emphasized  that  the  factors  listed  above  eombined  and 
acted  to  increase  the  aggregate  level  of  pereeived  eomplexity  and  cognitive  load 
for  eommanders  and  their  battle  staffs.  The  mission  command  role  itself  is 
intrinsically  complex  and  demanding.  However,  a  work  setting  with  a  large 
number  of  design-related  “rough  edges”  gives  the  impression  of  being  more 
complex  and  intimidating  than  one  that  is  better  designed  and  integrated  for 
effective  use.  Thus,  while  some  of  the  cognitive  workload  is  intrinsic  to  the  tasks, 
additional  workload  is  a  needless  consequenee  of  insuffieient  HSI.  The  laek  of 
training  and  familiarity  results  in  a  greater  need  to  cognitively  monitor  what  one 
is  doing  and  what  is  happening  (a  knowledge-based  activity).  With  adequate 
familiarization  and  training,  this  activity  becomes  skill-  or  rule-based,  which 
reduces  the  need  for  monitoring  and  frees  up  cognitive  channels,  thereby  reducing 
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cognitive  load  (Rasmussen  1983).  Errors  are  more  likely  if  skill  sets  are  knowledge- 
based.  Training  and  equipment  familiarity  are  important  considerations  in  pereeived 
eomplexity,  cognitive  load,  and  performance  reliability. 

3.  NIE  14.2  Data  Collection  and  Reporting 

ARE  HRED’s  HSI  work  for  the  BMC  CG  continued  into  NIE  14.2.  Prior  to  the 
start  of  NIE  14.2,  the  mission  command  complexity-cognitive  load  issue  was 
elevated  in  status  to  a  formal  Department  of  the  Army  (DA)  Objective.  Beeoming 
a  formal  DA  Objective  meant  that  the  issue  would  receive  additional  eommand 
emphasis  and  resourees,  and  that  analysis  results  would  be  ineluded  in  the  BMC’s 
Executive  Summary  of  NIE  results.  Analysis  results  and  reeommendations  also 
would  be  reported  in  a  separate  section  of  the  DA  Objeetives  Annex  report  for 
NIE  14.2. 

The  HSI  support  team  “eame  late”  to  the  formal  planning  proeess  for  NIE  14.2. 
Consequently,  the  team  carried  forward  the  objeetive  and  issues  used  in  the  BMC 
CG’s  CP  Ease  of  Use  assessment.  When  used  within  the  eontext  of  a  DA 
Objeetive,  analysis  issues  are  formally  referred  to  as  Priority  Questions.  HSI- 
related  results  from  NIE  14.2  are  organized  around  the  BMC  CG’s  Ease  of  Use 
objective  (now  a  DA  Objeetive)  and  the  4  subordinate  issues  listed  previously. 
Results  for  eaeh  of  these  Priority  Questions  are  now  addressed  in  turn. 

DA  Objective;  Evaluate  solutions  that  simplify  and  proteet  the  network  while 
enhancing  the  command  post  by  decreasing  its  physieal  eomplexity  and  eognitive 
burden  on  eommanders  and  staff. 

Priority  Question  1;  What  eognitive  systems  engineering  issues  impaet  the 
horizontal  and  vertieal  integration  of  mission  command  systems  within  CPs  as 
eonfigured  in  NIE  14.2? 

Cognitive  systems  engineering  is  a  specialty  discipline  under  systems 
development  that  addresses  the  design  of  soeiotechnieal  work  systems  (Hollnagel 
and  Woods  2005).  A  socioteehnieal  system  is  one  in  whieh  humans  provide 
essential  funetionality  related  to  deeiding,  planning,  eollaborating,  and  managing. 
Drawing  on  contemporary  researeh  and  insights  from  eognitive,  soeial,  and 
organizational  psyehology,  eognitive  systems  engineering  seeks  to  design  work 
systems  that  are  effeetive  and  robust.  The  focus  is  on  amplifying  human 
capabilities  to  perform  cognitive  work  by  integrating  technical  functions  with 
required  human  cognitive  proeesses  and  making  that  cognitive  work  more 
reliable.  Cognitive  systems  engineering  involves  aetivities  such  as;  1)  the  design 
of  human  interfaees  and  interactions,  2)  communication  systems  and  collaboration 
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methods,  3)  training  systems  and  methods,  4)  teams,  and  5)  management  systems 
and  methods.  A  CP  is  an  example  of  a  soeiotechnical  system  that  can  potentially 
benefit  from  the  application  of  cognitive  systems  engineering  principles  and 
practices. 

Commanders  and  their  staffs  tended  to  view  network-enabled  CPs  such  as  those 
used  in  NIE  14.2  as  “complicated  and  fragile”  (i.e.,  unreliable).  For  example,  a 
draft  briefing  addressing  NIE  14.2  emerging  results  prepared  for  the  Vice  Chief  of 
Staff  of  the  Army  (VCSA)  dated  28  May  2014  stated:  “Sending  graphics  from 
CPOF  [Command  Post  of  the  Future]  to  JBC-P  [Joint  Battle  Command-Platform] 
is  complicated  and  unreliable;  not  all  graphics  transfer  correctly.  When 
transmitted  from  JBC-P  to  NW  [NETT  Warrior],  graphics  must  be  broken  down 
into  smaller  packages  and  reassembled  by  the  receiver.”  Similar  comments 
occurred  frequently  in  the  NIE  database.  As  a  result,  CP  personnel  constantly 
have  to  “work  the  workarounds”  to  meet  mission  objectives.  Commanders  and 
staffs  routinely  commented  on  the  distracting  and  frustrating  impact  of  having  to 
manage  their  mission  command  equipment  suites  on  their  more  important  role  of 
“managing  the  fight.”  This  diversion  of  cognitive  resources  to  managing  mission 
command  equipment  suites  is  a  nuisance  task  that  has  significant  implications  for 
perceived  cognitive  load  and  overall  mission  command  performance. 

None  of  the  above  comments  should  come  as  a  surprise.  CP  component  design 
and  integration  have  not  yet  been  approached  from  an  overarching  system-of- 
systems  perspective.  Moreover,  rigorous  and  inclusive  cognitive  systems 
engineering  principles  and  practices  have  not  been  applied  during  their  design  and 
development.  The  draft  Command  Post  2025  Concept  of  Operations  (CONOPS) 
developed  by  the  US  Army’s  Mission  Command  Center  of  Excellence 
emphasized  this  point,  remarking  that: 

“Current  Army  command  posts  are  the  result  of  unsynchronized 
requirements  and  efforts  of  multiple  programs  and  are  not 
integrated  as  a  system.  [  .  .  .  ]  This  lack  of  synchronization  results 
in  a  level  of  complexity  that  cannot  be  overcome  by  the  unit’s 
organization,  people,  or  available  training  resources.” 

The  lack  of  an  overarching  system-of-systems  perspective  coupled  with  a  failure 
to  consider  the  CP  as  a  soeiotechnical  system  is  a  significant  contributor  to 
perceived  complexity  and  associated  cognitive  load. 

Army  HSI  efforts  have  traditionally  been  applied  at  the  individual  system  level, 
and  that  has  been  the  case  with  most  of  the  individual  equipment  items 
comprising  NIE  14.2  CPs.  What  has  not  been  adequately  addressed  is  evaluation 
of  HSI  issues  arising  out  of  the  relationships  between  Soldiers  and  technology. 
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not  just  at  the  individual  system  level,  but  also  at  the  team  and  organizational 
levels.  Some  of  the  most  demanding  aspeets  of  CP  operations  as  observed  during 
the  NIEs  are  emergent  properties  that  only  show  up  when  the  systems  eomprising 
the  CP  are  brought  together  and  eonfigured  in  a  partieular  way.  Emergence  is  a 
term  used  to  deseribe  system  behavior  that  might  not  be  observed  without  the 
situational  interaction  of  lower-level  components  or  contributors  (Vicente  2006). 
These  emergent  properties  might  not  show  up  in  an  isolated  assessment  of 
individual  mission  command  component  systems.  An  example  of  one  of  the 
emergent  aspects  of  CP  operations  is  knowledge  management  (KM)  on  the  part  of 
the  battle  staff.  Observed  KM  deficiencies  are  addressed  in  additional  detail  under 
Priority  Question  4. 

The  solution  to  the  problem  discussed  in  the  previous  paragraph  is  straightforward 
but  complex  in  application.  Overall  CP  design  must  be  approached  as  an 
integrated  platform  from  both  materiel  and  operational  perspectives  (i.e.,  CP-as-a- 
Platform).  The  materiel  acquisition  side  of  CP  design  and  fielding  must  be  better 
integrated  and  coordinated.  This  would  be  a  significant  step  toward  addressing  the 
“fragility”  comment  noted  previously.  From  an  operational,  user  experience 
perspective,  designers  must  apply  cognitive  systems  engineering  principles  and 
practices  as  the  CP  is  being  developed,  and  not  as  an  afterthought.  Considering 
the  CP  from  a  materiel,  “widgets  alone”  perspective  will  not  resolve  the  growing 
cognitive  load  problem. 

The  previous  paragraph  addresses  the  long-term  development  of  an  objective  CP- 
as-a-Platform.  What  about  the  short  to  midterm  evolutionary  development  of  CPs 
going  forward?  In  the  aftermath  of  ARL’s  CP  Ease  of  Use  assessment  conducted 
during  NIE  14.1,  the  BMC  CG  tasked  ARL’s  BMC  support  team  to  characterize 
the  following  3  aspects  of  CP  operations  in  terms  of  their  contribution  to 
complexity  and  cognitive  load:  1)  Developing  and  maintaining  a  Common 
Operating  Picture  (COP);  2)  Building  Operations  Orders  with  graphic  overlays; 
and  3)  Distributing  these  products  to  lower  echelon  units  and  dismounted  forces. 
Interestingly,  these  aspects  of  CP  operations  are  the  same  deficiencies  noted 
previously  in  the  extract  from  the  VCSA  briefing. 

The  ARE  team  was  not  able  to  fully  complete  this  action  prior  to  or  during  NIE 
14.2.  The  team  is  prepared  to  continue  this  analysis,  but  subject  matter  expert 
support  and  access  to  the  new  mission  command  equipment  items  are  required. 
Based  on  direct  observations  of  CP  operations,  interviews  with  commanders  and 
battle-staff  members,  and  NIE  14.2  database  entries,  the  problems  noted  in  the 
referenced  VCSA  briefing  likely  have  something  to  do  with:  1)  component 
features  and  their  integration  (subtle  incompatibilities  that  show  up  when 
components  are  put  together  and  used  as  a  set — emergence  again),  2)  lack  of 


9 


necessary  levels  of  user  and  team  proficiency,  plus  3)  interactions  between  the  2 
problem  categories.  As  one  NIE  14.2  database  entry  phrased  it,  “both  training  and 
materiel  shortcomings”  contribute  to  these  deficiencies.  Some  of  these  issues 
might  be  resolved  through  more  detailed  analysis  as  the  BMC  CG  requested  and 
that  the  ARE  team  is  prepared  to  continue.  Results  might  be  useful  in  1) 
suggesting  areas  for  enhanced  physical  integration  of  components,  2)  revisions  to 
CP  procedures  to  improve  operational  integration,  3)  improved  training  for 
individuals  and  teams,  and  4)  informing  specifications  for  a  next-generation 
integrated  CP. 

Priority  Question  2:  Assess  Soldier  NET  on  capability  understanding  and  its 
implementation  within  the  tactical  context  as  an  enabler  for  commander  decision 
making.  Provide  recommendations  for  beneficial  changes. 

NET  provides  for  the  initial  training  and  transfer  of  knowledge  from  the  Materiel 
Developer  or  vendor  to  testers  and  users.  It  represents  the  knowledge  that  is 
needed  for  operation,  maintenance,  and  logistic  support  during  testing  and  initial 
introduction  of  new  materiel  into  the  Army  inventory.  NET  is  intended  to  assist 
commanders  in  achieving  operational  capability  in  the  shortest  time  practical  by 
training  Soldiers,  crews,  and  maintainers  on  how  to  operate  and  maintain  new  or 
improved  equipment.  It  also  provides  unit  leaders  with  training  support 
components  needed  to  sustain  the  proficiency  of  operators  and  maintainers  of  the 
new  or  improved  equipment.  NET  is  provided  as  needed  prior  to  testing  and 
handoff  of  equipment  to  gaining  commands.  In  present  terminology,  operate 
should  not  be  confused  with  use  effectively  in  a  tactical  environment. 

Based  on  NIE  observations  and  user  reports,  it  is  obvious  that  NET  alone  is  not 
sufficient  to  prepare  the  unit  to  participate  meaningfully  in  an  NIE.  As  noted 
above,  NET  is  intended  to  provide  for  the  transfer  of  initial  system-related 
knowledge  from  the  developer  to  testers  and  first  users.  NET  is  not  intended  to 
serve  as  a  substitute  for,  or  to  be  considered  the  equivalent  of,  various  aspects  of 
follow-on,  unit-based  training  in  developing  mission-essential  levels  of 
individual,  crew,  team,  or  unit  proficiency.  Moreover,  the  NET  concept  arguably 
is  being  misapplied  prior  to  the  NIEs.  NET  is  an  orientation  and  familiarization  or 
“delta”  training  event;  it  is  not  an  adequate  substitute  for  full-spectrum  skill 
development  across  the  unit.  In  current  terminology,  a  delta  training  event  is 
intended  to  transition  a  potential  user’s  skill  set  from  one  component  to  another 
that  is  considered  an  upgrade  or  replacement.  It  is  assumed  that  users  are  fully 
proficient  in  the  use  of  predecessor  components  prior  to  attending  NET. 

In  addition  to  problems  with  NET  application,  there  is  inadequate  follow  up  to 
NET  within  the  unit.  NET  must  be  part  of  a  more  integrated  and  comprehensive 
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approach  to  pre-NIE  participant  preparation,  taking  into  account  the  “realities”  of 
both  skill  aequisition  and  retention.  This  is  particularly  true  for  critieal,  high-skill 
areas  such  as  network  operations  or  mission  command.  Moreover,  the  current 
“eheekerboard”  application  of  NET  prior  to  an  NIE  (i.e.,  fitting  NET  into  “gaps” 
in  the  unit’s  existing  training  calendar)  is  almost  guaranteed  to  result  in 
considerable  skill  deeay.  The  training  literature  is  clear  and  states  that  significant 
skill  decay  can  occur  within  relatively  short  timeframes  (days  to  weeks)  for  any 
form  of  skill  or  learned  material  (Hoffman  et  al.  2014b).  NET  resourees 
effeetively  are  wasted  when  signifieant  time  gaps  occur  between  participant 
training  and  event  performance,  unless  NET  is  followed  up  by  suitable  skill¬ 
reinforcing  training  activities.  In  the  case  of  the  NIEs,  the  gaps  between  training 
and  event  performance  can  be  as  long  as  6  months. 

NET  can  and  should  be  improved  to  make  it  more  effective  in  those  situations 
where  the  concept  is  appropriately  applied.  Potential  areas  for  NET  improvement 
inelude  the  categories  listed  below.  Reeommendations  coneeming  NET 
improvements  are  based  on  the  NET  Evaluation  Eramework  provided  in  the 
appendix.  In  present  context,  the  term  “framework”  refers  to  a  conceptual 
structure  for  evaluating  pre-exereise  NET  and  suggesting  potential  areas  for 
improvement. 

1)  Attention  to  NET  attendees  and  their  readiness  to  benefit  from  training: 

.  The  personnel  who  attend  NET  should  be  those  who  will  use  various 
equipment  items  or  groups  of  items  during  the  NIE; 

.  Attendees  should  be  “cleared”  for  NET  in  terms  of  essential  training 
and  experienee  prerequisites; 

.  Unit  leaders  must  not  be  allowed  to  exempt  themselves  from  NET; 
and 

.  NET  attendance  must  be  more  than  a  course  checkoff  or  seat-filling 
exereise. 

2)  Attention  to  NET  instruetional  processes: 

.  Improved  pretraining  sereening  to  ensure  that  proposed  NET 
conforms  to  sound  instructional  principles  and  praetices; 

.  Cheeking  to  ensure  that  NET  performanee  objectives  have  been 
identified  and  are  verifiable,  not  simply  topic-oriented; 
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.  Instructor  selection;  Participant  feedback  and  direct  observations 
suggest  that  the  best  NET  instruetors  address  both  technical  and 
tactical  aspects  of  equipment  use  (i.e.,  how  to  use  the  system  in  a 
tactical  context); 

.  Availability  of  eourse  referenee  materials  and  takeaways:  NET 
instructors  should  provide  attendees  with  referenee  materials  and 
instruetional  takeaways  to  reinforce  and  refresh  the  material  covered 
during  instruetion;  and 

.  NET  realism:  Realistic  expectations  about  how  much  actual  learning 
can  be  accomplished  during  the  time  alloeated  for  NET. 

3)  Insistenee  on  some  level  of  performanee  verification  as  a  NET  quality 

check: 

.  Require  knowledge  and  skill  checks  during  and  at  the  conclusion  of 
NET; 

.  Bear  in  mind  that  NET  is  primarily  eoncerned  with  “knowing  about,” 
with  a  lesser  emphasis  on  “knowing  how  to  do”; 

.  Appropriate  levels  of  “knowing  how  to  do”  must  be  developed  during 
follow-on  individual,  team/erew,  and  unit  training — usually  in  a  unit 
context. 

The  most  important  part  of  determining  the  extent  to  which  the  Soldier  “knows 
how  to  do”  is  the  development  of  objeetive  and  usable  performanee  measurement 
technologies.  Developing  sueh  measures  has  been  a  ehallenge  to  leaders  and 
trainers  for  decades  (Seibert  et  al.  2011).  Instead  of  relying  on  paper  notes  and 
memory,  raters  armed  with  observer-based,  scaled  performance  measures 
implemented  to  mobile  apps  (e.g.,  tablet  computers),  would  be  able  to  colleet  and 
record  real-time  performance  data  at  both  the  individual  and  team  level. 

Eacilities  exist  at  Eort  Bliss,  Texas  (e.g.,  the  1st  Armored  Division’s  Mission 
Training  Center  [MTC]  and  the  BMC’s  Mission  Command  Complex  [MCC])  to 
provide  some  of  what  is  necessary  to  ensure  that  participants  are  prepared  to 
perform  at  the  levels  required  by  the  NIEs.  However,  these  facilities  cannot 
provide  everything  that  the  unit  requires  for  adequate  pre-event  preparation.  Some 
essential  pre-event  training  can  only  be  eonducted  when  the  unit  has  access  to  the 
networks  and  the  actual  versions  of  the  various  mission  command  systems  that 
will  be  used  during  the  NIE.  As  one  foeus  group  participant  put  it,  “We  never  see 
the  full  fires  network  setup  until  we  get  out  to  the  box.” 
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In  summary,  the  following  points  are  concrete  recommendations  for  beneficial 
changes  to  pre-NIE  NET  and  other  aspects  of  participant  preparation: 


.  NET  must  be  based  on  a  solid  foundation  of  individual,  crew/team,  and 
unit  proficiency.  Even  “good”  NET  that  is  well  executed,  but  used  alone  is 
not  sufficient  to  achieve  this  goal.  The  unit  contends  that  it  does  not  have 
the  time  to  attend  necessary  training  beyond  NIE-related  NET  and  specific 
Army  Eorces  Command  (EORSCOM)-mandated  gunnery  and  other 
training  requirements. 

.  Verify  that  individuals,  crews,  and  teams  are  ready  to  participate 
meaningfully  in  NET.  Check  that  essential  training  and  experience 
prerequisites  have  been  met. 

.  As  currently  conceived  and  conducted,  NET  is  of  mixed  quality  and 
questionable  effectiveness.  NET  can  and  should  be  improved  to  be  more 
effective.  The  NET  Evaluation  Eramework  given  in  the  appendix  provides 
additional  detail. 

.  Component  NET  must  be  followed  by  appropriate  amounts  of  individual 
and  collective  sustainment  training  in  the  unit.  Not  addressing  this  need 
adds  to  the  Soldier  performance  risk  associated  with  skill  decay  and 
incomplete  or  inadequate  learning.  One  of  the  emerging  lessons  associated 
with  the  deployment  of  the  new  generation  of  CP  components  (CPOE, 
JBC-P,  etc.)  is  that  these  systems  require  training  well  beyond  what  is 
provided  during  NET.  Traditional  NET  typically  focuses  on 
“buttonology” — the  basics  of  system  usage  (and  sometimes  maintenance). 
Effective  use  of  these  systems  requires  a  deeper  understanding  of  overall 
system  capabilities  and  how  these  capabilities  support  command  decision 
making.  That  level  of  training  is  arguably  a  “bridge  too  far”  for  traditional 
NET.  Adequate  follow-on  training  is  also  essential  prior  to  team-oriented 
training  addressing  the  operations  of  the  CP-as-a-Platform.  Participants  in 
the  post-NIE  brigade  and  battalion  commander  focus  group  reinforced  this 
point,  noting  that,  “[We]  can’t  expect  Soldiers  to  go  through  40-50  h  of 
training  and  use  systems  efficiently;  [we]  need  [collective]  sustainment 
training.” 

•  The  unit  must  be  able  to  train  with  the  network  and  equipment 
configurations  they  will  actually  use  during  the  NIE.  There  is  some 
question  concerning  whether  adequate  time  is  available  immediately  prior 
to  an  NIE  (i.e.,  during  the  Integration  Motor  Pool  [IMP])  to  meet  this 
requirement.  Adequate  time  for  pre-event  training  is  a  serious  issue  and 
should  not  be  ignored  or  dismissed  with  the  claim  that  “things  have  always 
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been  this  way”  or  “inadequate  training  is  an  ‘NIEism’  that  we  just  have  to 
live  with.”  The  new  class  of  digital  mission  command  systems  is  less 
forgiving  of  user  performance  inadequacies  than  their  less  complex 
predecessors  (Hawley  1996).  Moreover,  from  a  practical  perspective, 
inadequate  participant  training  can  negatively  impact  the  validity  of  any 
inferences  derived  from  NIE  results  (Hawley  2007). 

Priority  Question  3:  Assess  Soldier  NET  from  a  “CP-as-a-System/Platform” 
perspective  and  its  implementation  within  the  tactical  context.  Provide 
recommendations  for  beneficial  changes. 

The  previous  discussion  emphasized  that  CP  component  design  and  integration 
have  not  been  approached  from  a  system-of-systems  or  cognitive  systems 
engineering  perspective.  The  lack  of  an  overarching  system-of-systems 
perspective  during  CP  design  along  with  a  failure  to  consider  the  CP  as  a 
sociotechnical  system  (battle  staff  plus  equipment),  is  a  significant  contributor  to 
perceived  complexity  and  associated  cognitive  load.  It  is  arguable  that  the  CPs 
observed  during  the  NIEs  have  not  been  explicitly  or  purposively  “designed”  in 
the  standard  use  of  that  term.  Rather,  they  consist  of  a  collection  of  loosely 
integrated  individual  systems  placed  in  the  CP  and  intended  to  support  the 
mission  command  warfighting  function.  Moreover,  each  of  the  individual 
components  has  been  developed  and  evaluated  in  isolation  as  a  single  system. 
Eittle  consideration  typically  has  been  given  to  how  these  systems  fit  together  as 
an  integrated  suite  to  support  mission  command. 

The  definition  of  a  system-of-systems  cited  earlier  notes  that  the  functionality  and 
performance  of  the  whole  is  more  than  the  simple  sum  of  the  component  systems. 
That  being  the  case,  one  cannot  adequately  evaluate  a  system-of-systems  by 
assessing  the  individual  components  in  isolation  and  then  rolling  up  the  results  to 
represent  the  whole  with  the  expectation  that  the  resultant  will  be  a  fully 
functioning  whole.  Eor  example,  the  CS-13  Culminating  Observations,  Insights, 
and  Eessons  report  from  4/10  Mountain  amplifies  this  point,  noting  that,  “There 
are  no  standalone  capabilities  within  CS-13”  (4-BCT,  10th  Mountain  Division 
2013).  Considering  the  CP  as  a  sociotechnical  system  requires  addressing  not  only 
component  design  and  integration,  but  also  issues  related  to  battle-staff  training, 
team  composition  and  structure,  and  CP  management  practices  (Standard 
Operating  Procedures  [SOPs];  Tactics,  Techniques,  and  Procedures  [TTP];  etc.). 

Pre-event  training  for  battle  staffs  across  the  brigade  was  not  approached  in  an 
integrated  manner.  The  same  criticism  leveled  at  CP  hardware  configurations  in 
the  CP  2025  CONOPS  could  also  be  made  with  respect  to  training  for  CP 
operations — component-focused,  unsyncronized,  and  conducted  by  different 
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proponents.  Complex  eognitive  work  sueh  as  that  performed  in  a  CP  is  teamwork. 
The  battle  staff  is  a  team  and  must  be  trained  as  a  team.  A  relevant  lesson  from 
team  training  researeh  is  that  even  a  team  of  experts  does  not  automatically 
translate  into  an  expert  team  (Hoffman  et  al.  2014b).  Contemporary  CP  operations 
involve  both  task  work  and  teamwork  skills.  Essential  teamwork  skills  cannot  be 
developed  through  individual  training  alone.  Some  of  the  difficulties  observed 
during  NIE  CP  operations  can  reasonably  be  attributed  to  the  lack  of  a  “crew- 
based  mentality”  when  structuring  and  training  the  battle  staff.  The  battle  staff 
cannot  be  treated  as  a  “pickup”  team  and  be  expected  to  perform  satisfactorily 
using  increasingly  complex  mission  command  equipment  suites. 

The  above  statement  is  particularly  true  of  the  battle  captain.  The  battle  captain 
plays  a  key  role  in  battle-staff  operations  and  in  transforming  data  and 
information  into  knowledge  that  a  commander  is  able  to  use  to  support  decision 
making.  The  battle  captain’s  role  can  no  longer  be  viewed  as  just  another  duty  to 
be  assigned  to  a  relatively  junior  officer  and  requiring  little  or  no  formal  training, 
follow-on  preparation,  or  performance  certification.  Eor  example,  the  HSI  support 
team  observed  many  instances  across  NIEs  in  which  a  battalion-level  battle 
captain  was  a  junior  lieutenant  having  little  or  no  formal  training  in  mission 
command,  the  MDMP,  the  equipment  used  to  support  a  battle  captain  (most 
frequently  CPOE),  or  any  of  the  supporting  mission  command  systems  used  in  the 
CP. 

A  team-based  approach  to  integrated  CP  NET  was  supposed  to  take  place  prior  to 
NIE  14.2 — the  Mission  Command  Systems  Integrated  Training  (MCSIT)  event. 
In  retrospect,  that  event  must  be  viewed  as  a  failure.  Participants  in  the  post-NIE 
brigade  and  battalion  commander  focus  group  characterized  the  MCSIT  event  as 
follows: 

“The  tools  the  trainers  brought  were  180°  from  what  we  needed  to 
do;  they  taught  in  PowerPoint;  training  needs  to  be  a  precursor  to 
NIE  scenarios;  i.e.,  [operations]  order  to  S-2  to  conduct  IPB 
[Intelligence  Preparation  of  the  Battlefield];  allow  us  to  build 
products  so  that  you  find  out  whether  you  can  push  graphics  from 
JBC-P  to  CPOE  and  to  other  ABCS  [Army  Battle  Command 
Systems]  systems;  trainers  are  not  integrated  and  have  their  own 
agenda.  They  only  taught  what  they  were  comfortable  teaching;  it 
was  a  step  back;  they  couldn’t  come  to  our  level.” 

It  is  arguable  that  the  MCSIT  event  alone — even  if  successfully  conducted — 
would  not  have  prepared  battle-staff  personnel  to  function  effectively  as  teams  or 
as  a  team-of-teams  across  the  brigade’s  command  echelons.  Brigade  and  battalion 
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leaders  remarked  that,  “The  only  training  [we  receive]  is  during  NET  .  .  .  there  is 
no  time  for  collective  training  .  .  .  this  BDE  doesn’t  have  the  ability  to  do 
sustain[ment]  training  because  of  constant  changes  to  software  and  systems.”  A 
caution  from  the  training  literature  is  relevant  to  this  last  comment:  Developing 
training  for  a  complex  task  is  itself  a  complex  task  (Hoffman  et  al.  2014b). 
Effective  training  for  a  complex,  team-based  activity  like  mission  command 
requires  special  expertise  and  resources  that  generally  are  not  available  in  a 
standard  Army  brigade.  Moreover,  there  is  more  to  the  conduct  of  effective 
individual  and  collective  training  than  access  to  adequate  training  equipment  (i.e.. 
Training  Aids,  Devices,  Simulators,  and  Simulations  [TADSS]).  More  than  50 
years  of  training  research  has  consistently  shown  that  training  design  and 
implementation  issues  generally  trump  issues  pertaining  to  training  equipment 
and  simulator  fidelity  (Salas  et  al.  1998).  As  these  authors  assert  in  the  title  of  the 
referenced  article,  “It  is  not  how  much  you  have,  but  how  you  use  it.”  The  “how 
you  use  it”  part  is  the  most  complex  aspect  of  the  task  of  developing  effective 
training  for  a  complex,  team-based  activity  like  mission  command. 

Given  the  growing  complexity  and  team-related  nature  of  contemporary  CP 
operations,  the  time  might  have  come  to  consider  CP  training  along  the  lines  of 
Crew  Resource  Management  (CRM)  for  commanders  and  their  battle  staffs.  Once 
almost  exclusively  an  aviation  program,  CRM  is  now  broadly  viewed  as  the  use 
of  all  available  human,  informational,  and  equipment  resources  toward  effective 
and  efficient  operations  in  operational  domains  dependent  on  crew  or  team 
performance  (Helmreich  et  al.  1999). 

In  summary,  the  following  points  are  concrete  recommendations  for  beneficial 
changes  to  pre-NIE  NET  and  other  aspects  of  participant  preparation  from  a  “CP- 
as-a-Platform”  perspective: 

.  Training  for  battle  staffs  across  the  participating  unit’s  command  echelons 
must  be  approached  in  an  integrated  manner,  matching  the  system-of- 
systems  nature  of  the  CP  itself. 

•  Battle  staffs  must  be  formally  structured  and  trained  as  a  team.  This  might 
require  battle  rostering  battle  staffs  and  performance  certifying  them  as  a 
team — as  is  currently  done  with  various  weapons  platform  crews,  such  as 
the  M-1  Abrams  or  M-2  Bradley.  Battle  rostering  battle  staffs  also  might 
help  to  alleviate  one  of  the  observed  problems  with  CP-oriented  NET: 
Who  attends?  In  the  case  of  the  battle  staff,  the  rostered  team  participates 
as  a  group. 

•  Special  training  and  development  attention  must  be  paid  to  the  battle 
captain  as  a  key  role  in  CP  operations:  the  orchestration  of  KM  processes 
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within  the  CP.  KM  skills  must  be  identified  and  explieitly  trained  in  a 
team  eontext.  During  focus  group  sessions,  the  unit  leadership  was 
adamant  that  they  must  conduct  this  training  themselves;  they  view  battle 
staff  training  as  a  unit  responsibility.  However,  it  is  not  clear  that  the  unit 
has  the  capability  to  adequately  develop  and  conduct  such  training. 

•  Leadership  and  senior  staff  at  all  levels  must  not  exempt  themselves  from 
battle-staff  team  training  events.  Commanders  at  all  echelons  must  learn  to 
work  with  their  supporting  battle  staffs  to  leverage  the  extensive  decision 
support  capabilities  that  network-enabled  mission  command  systems  now 
provide.  Moreover,  new  technology  can  change  the  nature  and  structure  of 
the  work  performed  in  CPs;  commanders  and  senior  staff  must  adapt  to 
these  changed  work  demands. 

•  The  battle  staffs  at  various  command  echelons  across  the  brigade  must 
also  learn  to  work  as  a  team-of-teams.  This  will  necessitate  more  extensive 
use  of  brigade-level  Command  Post  Exercises  (CPXs)  prior  to  an  NIE. 

Priority  Question  4:  Evaluate  complexity  and  cognitive  load  within  CPs  as 
configured  in  NIE  14.2.  Provide  specific  examples  of  sources  of  complexity  and 
cognitive  load  in  current  CP  operations.  Evaluate  cognitive  “pressure  points.” 
Provide  recommendations  to  reduce  complexity  and  cognitive  burden. 

The  underlying  theme  in  the  referenced  DA  Objective  is  managing  complexity. 
Prom  an  operational  mission  command  perspective,  NIE  results  strongly  support 
an  argument  that  “unmanaged”  CP  complexity  has  unnecessarily  added  to  the 
cognitive  load  on  commanders  and  their  battle  staffs.  Contemporary  CPs  might  be 
complex — because  a  complex  tool  is  often  necessary  to  do  complex  things — but 
that  tool  need  not  be  unnecessarily  complex.  The  consensus  of  literature  and 
experience  within  the  cognitive  systems  engineering  domain  is  that  managing 
complexity  is  a  partnership.  Designers  have  to  produce  products  that  help  users 
navigate  the  inherent  complexity  of  the  domain.  However,  users  of  these  products 
also  have  to  do  their  part.  Users  have  to  take  the  time  to  learn  the  structure  of  their 
new  tools  and  practice  the  skills  involved  in  their  effective  use  (Norman  2011). 
There  are  no  “silver  bullets”  to  overcoming  complexity  in  contemporary  CP 
operations.  Both  well-designed  components  and  system-of-systems  along  with 
well-designed  training,  delivered  in  appropriate  amounts,  are  the  solutions  to 
managing  complexity.  In  the  case  of  CPs  as  configured  in  NIE  14.2,  neither  of 
these  aspects  of  complexity  management  has  been  particularly  well  done. 

Providing  specific  examples  of  sources  of  complexity  and  cognitive  load  in 
current  CP  operations  is  consistent  with  a  methodology  routinely  used  in  applied 
human  factors  work  termed  the  critical  incident  technique  (Elanagan  1954). 
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Critical  incidents  are  ones  where  there  might  be  loss  of  life  or  property,  as  in 
aviation  aecidents.  The  eritical  incident  method  is  used  to:  1)  identify  and 
eharacterize  problems  in  the  relationship  among  humans  and  technology,  and  2) 
suggest  solutions  to  these  problems.  These  ehallenging  situations  are  sometimes 
ealled  eognitive  “pressure  points.”  With  respect  to  CPs,  cognitive  pressure  points 
are  plaees  in  the  mission  command  operational  flow  where  situational  demands  or 
CP  features  stymie  or  overwhelm  the  problem-solving  and  eoping  resourees 
available  to  the  battle  staff.  These  ean  inelude  pressure  points  resulting  from 
design  and  eomponent  integration,  maeroergonomic  (platform-level)  CP 
defieieneies,  or  battle-staff  expertise  defieieneies — knowledge,  skills,  and 
experience.  The  intent  of  the  assessment  is  to  explore  and  deseribe  how 
teehnologieal  ehanges  transform  eognitive  and  eollaborative  activities  and 
performanee  demands  within  the  CP  and  how  the  battle  staff  must  adapt  to  eope 
with  these  demands. 

The  eritieal  ineidents  diseussed  in  the  paragraphs  to  follow  are  deseribed  and 
evaluated  with  the  above  intent  in  mind.  These  eritieal  incidents  (Flanagan  1954; 
Seibert  et  al.  2011)  ean  beeome  the  foundation  of  benehmarked,  sealed  eolleetive 
performance  measures  for  training  assessment.  Prototype,  behaviorally  anehored 
rating  scales  are  eompatible  with  any  industrial  or  military  team-level  work 
environment  where  teams  must  eoordinate  on  eomplex  tasks. 

3.1  Information  Overload 

An  0/C  report  from  the  1st  Squadron,  1st  Cavalry  Regiment’s  CP,  dated  8  May 
2014  made  the  following  observation  regarding  the  Query  Tree  (QT)  widget 
(multiuse  tool)  used  to  support  intelligenee  analysis.  The  QT  widget  is  one 
eomponent  of  the  Distributed  Common  Ground  Station-Army  (DCGS-A)  system. 

“Users  are  unable  to  siphon  relevant  exereise  information  from  the 
Query  Tree  due  to  several  design-related  features  and  naming 
eonvention  SOPs  that  blend  or  hide  information  among  thousands  of 
real-world  SIGACT  [signifieant  activities]  entries.  .  .  .  Users  are  unable 
to  filter  [relevant]  information.  .  .  .  The  time  required  to  sift  through 
thousands  of  irrelevant  entries  makes  the  QT  widget  an  extremely 
ineffieient  tool  for  intelligenee  analysis.  .  .  .  The  inability  to  adequately 
loeate  relevant  information  quiekly  and  effieiently  in  the  QT  widget 
delays  S-2  analytics  beyond  the  point  of  operational  neeessity.” 

On  first  glanee,  the  QT  problem  deseribed  above  is  one  of  simple  information 
overload:  Too  mueh  information  is  provided  to  intelligenee  analysts  for  them  to 
proeess  in  a  timely  manner.  Consequently,  they  eannot  readily  separate  relevant 
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“signal”  from  background  “noise.”  This  also  could  be  a  problem  of  inadequate 
training  and  experience;  Analysts  simply  do  not  know  where  to  look  for  relevant 
data.  However,  the  0/C  noted  that  the  information  filtering  problem  reflects  an 
underlying  design  deficiency:  Naming  convention  SOPs  [that]  blend  or  hide 
relevant  information  among  thousands  of  entries.  Design  features  for  systems 
sueh  as  the  QT  widget  often  reflect  the  views  and  preferences  of  engineers  and 
programmers  and  not  user  needs  or  capabilities.  Menu  options  and  naming 
conventions  should  be  organized  in  ways  that  reflect  users’  needs  and 
expectations.  Users  must  then  be  adequately  trained  to  use  the  QT  tool  to  support 
S-2  analytics.  The  0/C  author  also  noted  that  the  eurrent  design  and  use  of  the  QT 
widget,  “delays  S-2  analytics  beyond  the  point  of  operational  neeessity.”  In  other 
words,  the  QT  widget  eannot  readily  be  used  to  inform  commander  decision 
making  in  the  CP. 

The  solution  to  problems  such  as  the  QT  issue  is  straightforward  and  involves 
work-centered  analysis  and  design  followed  by  rigorous  usability  testing  and 
performanee  valuation.  The  mantra  is:  Understand  what  is  going  to  be  done  with 
the  tool;  design  the  tool  to  support  that  work;  and  then  verify  that  the  resulting 
product  meets  initial  intent  and  is  usable  by  the  target  audience.  Put  the  user  and 
the  user ’s  work  needs  first  rather  than  fitting  users  to  the  tool  after  the  fact. 

3.2  CP  KM  Deficiencies 

The  following  remarks  are  extracted  from  an  0/C  Drop  Card  from  the  brigade  CP 
dated  14  May  2014,  along  with  a  supporting  comment  from  a  post-NIE 
command-level  focus  group: 

“The  problem  is,  how  does  the  TOC  [Taetieal  Operations  Center] 
manage  information  and  create  understanding?  .  .  .  The  BDE  [brigade] 

TOC  continues  to  struggle  analyzing  and  distributing  information  that 
flows  into  the  TOC.  .  .  .  This  is  a  systems  management  issue.  There  is 
not  any  cross  communication  of  intel  across  the  TOC  floor.  The  BDE 
S-2  may  collect  it  [intel  data],  but  the  [battle  captain]  is  not  populating 
anything  on  the  COP  to  establish  SA  [Situational  Awareness].  ...  I 
cannot  help  but  think  that  the  BDE  CDR  [commander]  is  frustrated 
with  his  intel  and  assisted  understanding  of  what  is  occurring  in  the 
battle  spaee.  .  .  .  The  BDE  is  pieeemealing  the  fight  [eaeh  mission 
command  component  within  the  CP  is  operating  semiautonomously 
with  little  overall  coordination  by  the  battle  staff].  The  network  is 
pushing  information,  but  the  TOC  is  being  overwhelmed  with 
information.  A  lack  of  information  management  from  the  network  is 
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causing  frustration.  .  .  .  Information  must  be  better  leveraged  to  make 
deeisions  more  rapidly.” 

The  following  is  an  independent  supporting  comment  from  a  post-NIE  foeus 
group: 

“We  have  the  teehnology/capability  but  ean’t  seem  to  figure  out  how  to 
do  it;  we  struggle  to  get  a  pieture  from  platform  to  platform.  How  do 
we  solve  this?  The  network  has  tremendous  capability  but  is  incapable 
of  being  leveraged  by  BDE  and  below  to  its  fullest  capacity;  for  4 
NIEs,  we  have  struggled  to  get  relevant  intelligenee  to  the  CO 
[Commanding  Offieer]  and  below  beeause  of  the  architecture.  .  .  .  We 
[require]  components  that  enable  staff  to  support  the  CDR 
intelleetually.” 

This  Drop  Card  is  an  insightful  comment  from  an  experieneed  0/C  characterizing 
the  roots  of  mission  command  dysfunction  in  the  brigade  CP  from  the  perspeetive 
of  complexity  and  eognitive  load.  Many  HSI  issues  are  implied  in  these  brief 
comments,  but  one  stands  out  in  partieular:  Inadequate  KM  skills  on  the  part  of 
the  battle  staff,  with  particular  reference  to  the  battle  captain.  KM  is  viewed  as  the 
process  by  which  data  are  transformed  into  information  (data  in  context),  and 
information  is  then  transformed  into  knowledge  that  can  be  used  to  support 
command  decision  making  (i.e.,  to  “support  the  CDR  intelleetually”).  The 
eognitive  processes  that  underlie  KM  include  colleeting,  organizing,  and 
summarizing  incoming  data  to  form  information  (data  in  context).  Information  is 
analyzed  and  synthesized  to  support  knowledge  “ereation.”  The  final  step  in  the 
KM  process  is  command  decision  making.  An  information  paper  on  training  for 
mission  command  produced  by  the  Mission  Command  Center  of  Exeellence 
characterizes  KM  as  the  “Binding  Idea”  underlying  effective  CP  operations 
(Training  the  Mission  Command  Warfighting  Eunction  2013).  The  eognitive 
demands  associated  with  KM  are  not  trivial,  particularly  in  a  data-intensive  CP 
setting. 

The  implieations  of  the  previous  comments  are  clear.  KM  was  not  being 
performed  effectively  in  the  brigade  CP.  Consider  eritical  supporting  remarks  in 
turn:  The  network  is  pushing  information  (data);  the  TOC  is  overwhelmed  with 
information;  the  brigade  battle  captain  is  not  populating  anything  on  the  COP  to 
establish  SA  (data  are  not  being  organized  and  summarized  into  usable 
information;  information  analysis  and  synthesis  are  not  being  performed);  a  lack 
of  information  management  from  the  network  is  causing  frustration.  And  finally  a 
capstone  remark  from  the  brigade  commander:  “Eor  4  NIEs,  we  have  struggled  to 
get  relevant  intelligence  to  the  CO  and  below  because  of  the  arehiteeture  .  .  .  We 
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[require]  components  that  enable  staff  to  support  the  CDR  intellectually.”  The 
issue  here  involves  more  than  just  the  network  and  CP  architecture  (CP 
component  design,  internal  organization,  and  physical  integration).  As 
emphasized  in  previous  sections,  NIE  14.2  CPs  were  not  explicitly  designed  to 
support  a  cognitively  dominated  activity  such  as  mission  command  in  an 
integrated  manner.  KM  is  an  important  aspect  of  effective  mission  command. 
However,  the  activities  comprising  KM  do  not  appear  to  have  been  adequately 
operationally  defined  and  integrated  into  routine  CP  procedures  or  mission 
command  processes.  Moreover,  battle  staffs  were  not  explicitly  trained  in  KM 
skills.  KM  is  a  team  activity,  and  the  battle  staff  was  not  trained  as  a  team. 

In  a  complex  sociotechnical  work  system  like  those  created  by  all  the  new 
technology  on  display  in  NIE  CPs,  one  cannot  just  throw  Soldiers  into  a 
transformed  workplace  (with  inadequate  training  and  experience — plus  all  the  old 
organizational  and  operating  concepts)  and  expect  them  to  succeed.  But  that  is 
just  what  we  do — and  expect  that  somehow  “the  Soldiers  will  make  it  work.” 
They  might  eventually  make  it  work,  but  they  struggle.  One  should  not  be 
surprised  by  what  is  continually  observed  and  reported  during  the  NIEs.  As 
Norman  (2011)  asserted,  complexity  must  be  explicitly  managed.  In  the  case  of 
NIE  CPs,  that  has  not  been  done  adequately,  if  at  all. 

4.  Discussion:  Mission  Command  Complexity  and  Cognitive 
Load  Going  Forward 


,  ,  ground  combat  is,  without  any  doubt,  the  most  complex  set  of 
interactions  in  any  kind  of  military  operations  by  any  service  or  any  country. 
Unfortunately,  just  asserting  this  does  not  help  much,”  General  William 
DePuy,  1975 

As  noted  in  the  previous  section,  2  problematic  follow-on  issues  fall  out  of  ARE 
HRED’s  HSI  work  performed  at  the  system-of-systems,  team,  and  organization 
levels  during  the  EY  2014  series  of  NIEs.  These  are:  1)  physical  and  operational 
integration  of  CP  component  systems,  and  2)  KM  by  the  battle  staffs  across  the 
brigade’s  command  echelons.  Results  from  the  NIEs  as  reported  herein  and 
elsewhere  support  the  observation  that  contemporary  Army  CPs  are  the  result  of 
unsynchronized  efforts  of  multiple  Program  Managers  and  are  not  integrated  as  a 
system.  As  such,  CPs  are  both  nonstandard  and  complex  from  physical  and 
cognitive  perspectives.  Physically,  CPs  require  significant  time  to  setup  and  tear 
down,  involve  extensive  wiring,  and  are  transit-case  dependent  for  mobility 
(Command  Post  2025  CONOPS  2014).  NIE  results  also  indicate  that  current  CPs 
are  complex  from  a  cognitive  perspective — complicated  and  difficult  to  use. 
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Norman  (2011)  used  the  term  eomplieated  to  mean  “puzzlingly  eomplex.”  It  has 
been  observed  that  the  unit  “struggles  to  fight”  using  the  mission  eommand 
systems  in  CPs  as  eonfigured  in  the  NlEs  (Hawley  2014). 

The  current  Army  approach  to  CP  development  and  procurement  has  inadequately 
addressed  the  integration  of  disparate  programs  and  materiel.  What  CP  integration 
there  is  has  been  approached  primarily  from  a  physical  perspective  and  has  not 
adequately  addressed  operational  integration  of  new  materiel  into  routine  mission 
command  processes.  As  a  result,  current  CPs  do  not  readily  support  mission 
command  as  cognitive  work.  That  is,  inadequate  physical  integration  and 
inattention  to  operational  integration  result  in  a  poor  level  of  perceptual-cognitive 
integration  that,  “cannot  be  overcome  by  the  unit’s  organization,  people,  or 
available  training  resources”  (Command  Post  2025  CONOPS  2014,  p.  3). 

Achieving  effective  operational  integration  often  necessitates  an  iterative,  “test- 
and-leam”  process  making  use  of  cognitive  systems  engineering  principles  and 
practices.  These  principles  and  practices  are  well  enough  understood  to  be  applied 
in  the  formulation  and  development  of  the  next-generation  of  Army  CPs,  as  well 
as  eliminating  the  worst  of  the  “rough  edges”  encountered  in  current  CPs. 
Cognitive  systems  engineering  principles  and  practices  are  referred  to  by  several 
names.  These  include  human-centered  design,  decision-centered  design,  and 
work-focused  design  (e.g.,  see  Lintem  2012).  The  idea  is  the  same:  CP 
component  integration  must  be  considered  from  the  point  of  view  of  the  team 
assigned  to  perform  mission  command  using  that  equipment  suite.  Based  on 
results  across  NlEs,  it  is  obvious  this  requirement  has  not  been  addressed. 

Lindell  et  al.  (2003)  define  what  they  term  a  “Hierarchy  of  Design  Needs”. 
Elements  of  this  hierarchy  are  illustrated  in  the  table.  These  authors  assert  that  for 
a  work  system’s  design  to  be  successful,  it  must  meet  user’s  basic  needs  before  it 
can  be  used  to  satisfy  higher-level  needs.  In  the  case  of  the  CPs  observed  during 
NIE  14.2,  it  is  not  clear,  when  viewed  as  an  integrated  system-of-systems  (i.e., 
CP-as-a-Platform),  that  requirements  for  the  first  level  in  the  hierarchy 
(Eunctionality)  have  been  fully  met.  Beyond  that,  NIE  results  strongly  suggest 
that  Reliability,  Usability,  Proficiency,  and  Creativity  needs  remain  unmet. 
Eurthermore,  Proficiency  and  Creativity  assume  that  users  are  skilled  in  the  use  of 
CP  equipment  items  and  in  conducting  mission  command  operations  using  that 
equipment.  NIE  results  cast  doubt  on  the  validity  of  that  assumption. 
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Table  Hierarchy  of  design  needs  (after  Lindell  et  al.  2003) 


Functionality 

The  work  system  meets  functional  requirements. 

The  work  system  exhibits  stable  and  consistent 

Reliability 

performance.  If  the  work  system  performs  erratically,  or  is 
subject  to  frequent  failure,  reliability  needs  are  not 
satisfied. 

The  work  system’s  technologies  are  forgiving  and  easy  to 

Usability 

use.  If  difficulty  of  use  is  too  great,  or  the  consequences  of 
simple  errors  too  severe,  usability  needs  are  not  satisfied. 
Proficiency  involves  empowering  users  to  do  things  better 

Proficiency 

than  they  could  previously.  (In  the  case  of  NIE  CPs,  this 
has  not  been  empirically  demonstrated.) 

All  prior  user  needs  have  been  satisfied,  and  users  begin 
interacting  with  the  work  technologies  in  innovative  ways. 

Creativity 

The  technology  can  now  be  used  to  create  and  explore 
areas  that  extend  both  the  technology  and  the  persons  using 
that  technology.  (Not  achieved  during  the  NIEs) 

Beyond  problems  resulting  from  inattention  to  physical  and  operational 
integration,  the  types  of  operations  that  the  US  Army  has  been  engaged  in  during 
the  past  decade  or  more  have  led  to  a  degradation  of  the  skills  required  to 
establish  CPs  and  conduct  mission  command  in  support  of  expeditionary  and  full- 
spectrum  mission  operations  (Command  Post  2025  CONOPS  2014).  Results  from 
domains  related  to  maneuver  unit  mission  command  indicate  that  widespread  use 
of  information  and  communications  technologies  changes  the  nature  of  the 
cognitive  work  those  technologies  are  intended  to  support.  And  these  changes 
impact  training  and  development  requirements  of  personnel  performing  that  work 
(Hawley  and  Mares  2012;  Hoffman  et  al.  2014a).  The  discussion  of  CP  problems 
in  the  previous  section  alludes  to  KM  “breakdowns” — arguably  attributable  to 
battle-staff  sensemaking  deficiencies  stemming  from  poor  operational  integration 
of  CP  components  and  inadequate  training  and  development  of  the  battle  staff.  It 
is  not  possible  to  make  sense  of  things  that  are  not  easy  to  understand,  regardless 
of  the  power  and  sophistication  of  the  available  technical  facilities.  In  the  words 
of  the  brigade  leadership  (cited  previously),  commanders  and  their  battle  staffs 
often  struggle  to  form  an  integrated  picture  or  mental  model  of  the  battlespace 
using  current  mission  command  components.  Adequate  perceptual-cognitive 
integration  remains  elusive  in  NIE  CPs. 

The  Army  formally  defines  KM  as  the,  “art  of  creating,  organizing,  applying,  and 
transferring  knowledge  to  facilitate  situational  understanding  and  decision¬ 
making"  (Field  Manual  [FM]  6-01.1  2012).  KM  supports  improving 

organizational  learning,  innovation,  and  performance.  KM  processes  ensure  that 
knowledge  products  and  services  are  relevant,  accurate,  timely,  and  useable  to 
commanders  and  decision-makers  (FM  3-0  2012).  It  creates  value  for 
organizations  by  increasing  operational  effectiveness,  decision  quality,  and  unit 
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innovation.  As  noted  previously,  a  White  Paper  on  mission  eommand  training 
prepared  by  the  Mission  Command  Center  of  Excellenee  refers  to  KM  as  the 
“Binding  Idea”  in  CP  operations  (Training  the  Mission  Command  Warfighting 
Function  2013). 

Leistner  (2010)  characterizes  KM  as  the  process  of  transitioning  from  data  to 
information,  where  information  is  defined  as  data  in  context.  Data  in  context  are 
then  translated  into  knowledge  that  can  be  used  to  support  command  decision 
making.  A  number  of  KM  theorists  and  practitioners  caution,  however,  that  the 
transformation  process  from  data  to  information  and  from  information  to 
knowledge  is  a  cognitive  activity  (Leistner  2010;  Walker  et  al.  2009;  Wilson 
2002).  Wilson  (2002,  p.  2)  bluntly  states  that,  “Knowledge  is  defined  as  what  we 
know:  knowledge  involves  the  mental  processes  of  comprehension, 
understanding,  and  learning  that  go  on  in  the  mind  and  only  in  the  mind,  however 
much  they  involve  interaction  with  the  world  outside  the  mind,  and  interaction 
with  others.” 

Walker  et  al.  (2009)  provide  a  related  perspective  on  the  cognitive  underpinnings 
of  the  process  of  moving  from  data  to  information  to  knowledge.  These  authors 
note  that  “live”  network-enabled  operations  involve  far  more  than  simply  having 
a  lot  of  data.  Effective  mission  command  requires  that  data  be  transformed  into 
required  information  in  a  timely  manner.  Moreover,  creating  information  from 
data  is  complicated  by  the  fact  that,  like  beauty,  what  is  considered  information  is 
largely  in  the  eye  of  the  beholder.  These  authors  go  on  to  observe  that  data 
(objective,  measurable  realities  of  a  situation)  provide  the  basis  for  Endsley’s 
(1995)  Level  1  SA:  the  perception  of  elements  in  the  environment.  Endsley’s 
Level  2  SA  refers  to  comprehension  of  what  those  elements  might  mean.  In  the 
eye  of  the  beholder,  data  are  becoming  information  (data  in  context).  Level  3  SA 
refers  to  mental  projection  into  the  future:  actually  doing  something  with  that 
information.  Grouped  together.  Level  2  and  Level  3  SA  define  categories  that  are 
more  “information-  and  knowledge-like”  than  they  are  “data-like”  (or  Level  1 
SA-like). 

Information  and  knowledge  are  the  products  of  individual  and  collective 
sensemaking  activities  within  the  CP.  Sensemaking  is  a  cognitively  centered 
activity.  Knowledge  creation,  the  final  step  in  the  KM  process,  very  much 
involves  attention  to  the  “cognitive  preparation” — ^background,  preparation,  and 
experience  level — of  the  battle-staff  personnel  responsible  for  transforming  data 
into  information,  and  information  into  knowledge  to  support  command  decision 
making. 


24 


In  a  recent  McKinsey  Quarterly  article,  Dewhurst  and  Wilmott  (2014,  p.  1) 
remarked  that  the  advances  of  so-called  “brilliant”  machines  in  supporting  senior 
leaders  might  “astound  us,”  but  only  if  those  leaders  “enable  them  to.”  They 
further  asserted  that  technology  may  require  a  transformation  of  the  concept  of  the 
senior  leader’s  role.  The  senior  leader’s  role  increasingly  will  focus  on:  1)  asking 
the  right  questions,  2)  attacking  exceptions,  3)  tolerating  ambiguity,  and  4) 
employing  “soft”  leadership  skills.  These  authors  went  on  to  observe  (p.  2)  that: 

“Today’s  unaware  leader  risks  drowning  in  minutia  .  .  .  Some  are 
already  reacting  [to  this  problem]  by  distancing  themselves  from 
technology — for  instance  by  empowering  layers  of  staffers  to 
screen  data,  which  gets  turned  into  more  easily  digestible 
PowerPoint  slides.  In  doing  so,  however,  [leaders]  risk  getting  a 
‘filtered’  view  of  reality  that  misses  the  power  of  the  data  available 
to  them.” 

HRED’s  HSI  support  team  has  observed  numerous  instances  of  this  situation 
across  the  brigade’s  echelons  over  the  course  of  4  NIEs.  A  prime  example  of  such 
“distancing”  is  the  Digital  Executive  Officer  (XO)  concept  observed  in  previous 
NIEs  and  discussed  in  Hawley  (2014). 

Increased  use  of  information  automation  is  often  presented  as  a  potential  solution 
to  information  overload  and  KM  problems  in  contemporary  CPs.  Information 
automation  refers  to  automating  data-  and  information-handling  activities  in  fast- 
paced,  information-rich  settings  such  as  a  CP  (Billings  1996).  Under  such  an 
approach,  various  software  applications  would  be  used  to  assist  the  battle  staff  in 
performing  KM  processes.  Information  automation  has  the  potential  to  be 
beneficial  in  many  ways,  and  enable  CP  operations  that  would  otherwise  be 
difficult  to  support.  However,  caution  is  suggested  because  information 
automation  can  have  negative  side  effects  if  not  implemented  properly.  Eor 
example,  a  recent  Eederal  Aviation  Administration  (EAA)  report  notes  that 
information  automation  improperly  applied  in  automated  flight  management 
systems  can  increase  pilot  workload,  increase  head-down  time,  distract  the  flight 
crew  from  higher  priority  tasks,  and  contribute  to  crew  communication  and 
coordination  issues  (EAA  2013).  Moreover,  control  vulnerabilities  can  occur, 
especially  if  users  are  not  aware  of  assumptions  made  in  the  support  system’s 
design  or  if  the  information  presented  to  users  is  not  fully  understood.  Used  in  this 
context,  the  term  vulnerability  refers  to  characteristics  or  issues  that  render  a 
system  or  process  more  likely  to  break  down  or  fail  when  faced  with  unusual  or 
ambiguous  situations.  These  cautions  from  the  flight  management  arena  are 
relevant  to  providing  automated  support  for  the  KM  activities  that  take  place 
within  contemporary  CPs. 
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The  word  “automation”  has  often  eome  to  be  an  unfortunate  eode  word  for  “fewer 
personnel,”  “less  expertise,”  and  “lower  eost”  (Hoffman  et  al.  2014a).  In  other 
words,  there  is  a  widespread  belief  in  some  portions  of  the  system  development 
eommunity  that  automation  is  a  ready  means  to:  1)  reduee  the  number  of  people 
in  a  given  system,  2)  reduce  the  amount  of  training  and  experience  these 
individuals  and  teams  require,  and  3)  lower  the  costs  associated  with  developing 
and  operating  that  system.  Results  from  across  a  number  of  sociotechnical  work 
domains  indicate  that  this  so-called  “substitution  fallacy”  is  one  of  the  “deadly 
myths”  associated  with  automation  (Bradshaw  et  al.  2013).  The  substitution 
fallacy  refers  to  a  naive  belief  among  some  system  developers  that  automation 
simply  shifts  tasks  from  users  to  the  machine,  and  a  consideration  of  how  users 
interact  with  that  automated  support  is  not  necessary  (Hollnagel  and  Woods 
2005).  The  prevailing  view  here  might  be  stated  as,  “Just  automate  that  function 
and  we  will  get  it  off  our  plate.”  Decades  of  human  factors  research  and  field 
experience  strongly  contradict  this  view  (Bradshaw  et  al.  2013;  Hoffman  et  al. 
2014a).  Information  automation  alone  is  not  likely  to  be  a  panacea  for  the  KM 
deficiencies  noted  in  contemporary  CPs.  Improperly  applied  or  “clumsy” 
information  automation  might,  in  fact,  make  the  KM  problem  in  CPs  worse,  as  the 
FAA  results  caution. 

One  HSI-related  insight  from  the  now  cancelled  Future  Combat  Systems  (FCS) 
program  was  that  without  significant  changes  in  personnel-  and  training-related 
concepts  and  practices,  the  Army  would  struggle  to  support  the  materiel  side  of 
that  acquisition.  Old  practices  in  these  areas  would  not  prove  sufficient  to  support 
the  new  warfighting  concepts  and  their  enabling  technologies.  In  many  respects, 
the  concepts  and  equipment  demonstrated  and  evaluated  during  the  NIEs  are 
conceptual  descendants  of  the  concepts  underlying  the  FCS  program.  Viewed 
holistically,  the  results  in  this  report  support  that  insight  from  the  FCS  program.  It 
is  true  that  the  unit  often  “struggles  to  fight”  with  immature  and  unreliable 
equipment  that  is  often  poorly  integrated  from  a  physical  perspective.  Problems  in 
these  areas  are  eventually  resolved.  A  potentially  more  intractable  set  of  issues 
concern  operational  integration  deficiencies  along  with  obsolete  personnel  and 
training  concepts  and  practices.  Policies  and  practices  in  the  personnel  and 
training  domains  will  have  to  be  evaluated  and  adjusted  in  light  of  lessons  from 
the  NIEs  and  the  deployment  and  use  of  CS  equipment  across  Army  units.  It  is 
often  said  that  information  and  communications  technologies  are  “skill-biased.” 
Widespread  use  of  these  technologies  in  a  work  setting  has  been  observed  to  raise 
the  skill,  knowledge,  and  aptitude  requirements  associated  with  their  effective 
use.  Training  and  personnel  practices  must  be  revised  to  reflect  this  new 
performance  context.  This  challenge  must  be  met  or  the  Army  will  fail  to  exploit 
the  potential  of  new  technologies.  As  the  brigade  commander  quite  accurately 
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stated  after  NIE  14.2,  “We  have  the  technology/capability  but  ean’t  seem  to  figure 
out  how  to  do  [use]  it.  The  network  has  tremendous  capability  but  is  incapable  of 
being  leveraged  by  BDE  and  below  to  its  fullest  capacity.”  A  major  aspect  of  the 
solution  going  forward  involves  increased  attention  to  cognitive  systems 
engineering  concepts  and  practices.  These  include:  1)  Well-designed  and 
appropriately  integrated  CPs,  and  2)  well-designed  training  delivered  in 
appropriate  amounts.  Moreover,  there  are  no  technical  silver  bullets.  Materiel 
solutions  alone  will  not  solve  the  growing  complexity-cognitive  load  problem  in 
CP  operations. 
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Appendix.  New  Equipment  Training  (NET)  Evaluation  Framework 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Background 

•  A  Quality  Assurance  framework  for  assessing  NET  effectiveness  and  pre¬ 
test  training  adequaey 

•  Assessment  components 

Input;  Trainees  and  their  readiness  to  attend  NET 
Process:  What  goes  on  during  NET 

Output;  Job-related  skills  and  knowledge  aequired  during  NET 

•  The  goal  is  to  make  NET  evaluation  evidenee-based  rather  than  mostly 
subjective 
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Input 

Trainee  eharaeteristies:  Do  we  have  the  right  target  audienee,  and  are  they 
properly  prepared  to  benefit  from  NET? 

1 .  MOS(s)  Awarded 

a.  When? 

2.  Time  in  Serviee? 

3.  Time  in  Grade? 

4.  Rank 

5.  Level  of  performance  certifications  obtained? 

a.  Table  IV,  Table  VIII,  etc. 

b.  Software  builds  qualified  on?  (PDB  6,  6.5,  7) 

c.  Date  of  last  certification? 

6.  Months  of  unit  time? 

a.  Units? 

b.  Roles  performed? 

c.  Time  by  job  position 

The  goal  is  to  determine  the  level  of  skills,  knowledge,  and  experience  Soldiers 
bring  to  NET. 
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Process 

NET  instructional  evaluation  criteria:  Do  NET  processes  meet  generally  accepted 
guidelines  for  effective  instruction? 

1.  Is  the  focus  of  NET  on  job  performance? 

2.  Are  tasks  and  skills  (Performance  Objectives)  identified? 

-  Are  NET  Performance  Objectives  job-  or  role-related? 

3.  Is  practice  realistic  and  job-related? 

4.  Does  practice  include  a  mix  of  job  situations? 

5.  Do  trainees  practice  to  job  competence  (the  level  required  by  the  test)? 

-  Have  measurable  task-  and  job-related  standards  been  specified? 

-  Is  allocated  time  for  NET  sufficient  to  meet  job-related 
performance  standards,  given  pre-NET  trainee  skill  levels  and 
experience? 

-  Does  NET  include  a  focus  on  whole-job  performance? 

-  How  much  time  will  elapse  between  the  end  of  NET  and  the  onset 
of  testing?  —  Will  trainees  have  an  opportunity  to  train  in  the 
interim? 

6.  Does  hands-on  practice  equal  at  least  half  of  training? 

7.  Instructor  qualifications:  Technical  experience?  Tactical  experience? 
Source:  Whitmore  (2002). 


34 


Output 

NET  quality  checks:  Evidence-based  versus  subjective  assessment  of  NET 
effeetiveness. 

•  Quality  is  defined  in  terms  of  job-related  knowledge  and  skills  aequired 
during  NET 

-  Assessed  in  terms  of  the  “delta”  between  what  the  Soldiers  brought 
to  the  NET  situation  versus  their  performanee  eapabilities 
following  NET 

•  Quality  eheek  eomponents: 

-  1 .  Trainee  questionnaire 

•  Caveat:  Trainees  often  don’t  know  what  they  don’t  know, 
and  won’t  know  until  ealled  upon  to  perform  in  a  taetieal 
situation 

-  2.  Knowledge  test  (knowing  about) 

-  3.  Task-related  skill  eheeks  following  training  bloeks  (knowing 
how  to  perform  individual  tasks) 

-  4.  End-of-NET  whole-job  skills  assessment  (knowing  how  to 
integrate  tasks  on  the  job) 

•  Condueted  within  the  eontext  of  a  standardized  post-NET 
exercise 

•  Can  the  Soldier-trainee  perform  tasks  within  the  job/role 
eontext? 
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Summary 

•  Evidence-based  versus  subjective  assessment  of  NET  effectiveness 

•  Consistent  with  generally  accepted  criteria  for  evaluating  course 
implementation  (Whitmore  2002) 

•  Consistent  with  Kirkpatrick’s  Eour-Eevel  Training  Evaluation  Model 
(Kirkpatrick  and  Kirkpatrick  2006) 

-  Reaction;  Did  trainees  like  the  course  and  think  it  was  useful? 

•  Trainee  post-NET  questionnaire 

-  Learning;  Did  trainees  learn  the  target  skills  and  knowledge? 

•  Knowledge  tests 

•  Skill  checks 

-  Transfer;  Can  trainees  apply  these  skills  and  knowledge  on  the 
job? 

•  Post  NET  whole-job  performance  assessment 

-  Results;  Did  it  impact  the  organization’s  bottom  line — improved 
unit  performance? 

•  Not  assessed  objectively 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


ABCS 

Army  Battle  Command  Systems 

ARL 

US  Army  Researeh  Faboratory 

ATEC 

Army  Test  and  Evaluation  Command 

BMC 

Brigade  Modernization  Command 

CG 

Commanding  General 

CO 

Commanding  Officer 

CONOPS 

Concept  of  Operations 

COP 

Common  Operating  Picture 

CP 

Command  Post 

CPOF 

Command  Post  of  the  Future 

CPX 

Command  Post  Exercise 

CRM 

Crew  Resource  Management 

CS 

Capability  Set 

DA 

Department  of  the  Army 

DCGS-A 

Distributed  Common  Ground  Station-Army 

FAA 

Federal  Aviation  Administration 

FCS 

Future  Combat  Systems 

FM 

Field  Manual 

FORSCOM 

Army  Forces  Command 

FY 

Fiscal  Year 

HRED 

Human  Research  and  Engineering  Directorate 

HSI 

Human-System  Integration 

IMP 

Integration  Motor  Pool 

IPB 

Intelligence  Preparation  of  the  Battlefield 

JBC-P 

Joint  Battle  Command-Platform 
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KM 

knowledge  management 

MANPRINT 

Manpower  and  Personnel  Integration 

MCC 

Mission  Command  Complex 

MCSIT 

Mission  Command  Systems  Integrated  Training 

MDMP 

Military  Decision  Making  Process 

MTC 

Mission  Training  Center 

NET 

New  Equipment  Training 

NIE 

Network  Integration  Evaluation 

NW 

NETT  Warrior 

0/C 

Observer/Controller 

QT 

Query  Tree 

SA 

Situational  Awareness 

SOP 

Standard  Operating  Procedure 

SoSI&E 

System  of  Systems  Integration  and  Engineering 

TADSS 

Training  Aids,  Devices,  Simulators,  and  Simulations 

TOC 

Tactical  Operations  Center 

TTP 

Tactics,  Techniques,  and  Procedures 

VCSA 

Vice  Chief  of  Staff  of  the  Army 

XO 

Executive  Officer 
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